Introduction to Modeling
Forrest W. Crawford
Overview
What is a model?
A model is a mathematical relationship that comes with a story. Stokey and Zeckhauser (1978) give a definition: “A model is a simplified representation of some aspect of the real world, sometimes of an object, sometimes of a situation or a process”.
A good model reduces a complex situation to a set of essential mechanisms, or dynamics, that an analyst needs in order to make a good decision.
A bad model mischaracterizes the mechanism of interest, is too simple to capture important dynamics, or is too complicated to be calibrated or understood.
Basu and Andrews (2013)
Good models
All models are wrong, but some are useful. – George Box
A good model is suited to a particular problem, and balances parsimony and realism, simplicity and complexity.
Parsimony and complexity: models of DNA
Models are used to answer hard questions
Sometimes the corresponding empirical study may be infeasible or unethical to conduct in real life.
For example,
What would happen if every injection drug user had access to naloxone? How many fatal overdoses would be averted? Is this intervention program cost-effective?
What would happen if the government eliminated funding for smoking cessation programs?
Modeling and scientific hypotheses
Models formalize scientific hypotheses about the mechanism that produces a phenomenon of interest.
When data agree with our model, then we may accumulate evidence that the model is correct, or at least that the data do not falsify the model.
When we observe data that do not agree with the predictions of our model, then this might be evidence that our hypotheses are wrong.
Model fit to empirical data
Observing that a model fits data well is not a sufficient condition to imply that the model is correct.
What do we mean “correct”? We mean mechanistic or causal. This goes beyond fitting data well. We mean that a model captures the mechanistic features of the data-generating process that are important for the decisions we want to make.
And, sometimes there are no data! For many of our most pressing public health decision-making challenges, no suitable data exist, and we need to invent a plausible model to evaluate the effects of policy possibilities.
Examples
Example: ART
Which of these questions should be addressed by modeling?
- What would be the difference in life expectancy for a patient who is prescribed ART at the time of HIV diagnosis vs. a CD4 count of 500?
- What would be the difference in HIV mortality rates in South Africa over the next 5 years if the ART guidelines switched to recommend prescribing ART for patients at the time of HIV diagnosis vs. when their CD4 count drops <500?
Examples: HIV models
Examples: broader public health modeling challenges
- Optimizing the HIV care pipeline
- Counting drug users and other risk groups
- Deciding whom to vaccinate against disease
- Stopping infectious disease outbreaks
- Optimizing hospital staffing
How to do it
A recipe for modeling
- State the thing you want to learn about
- State what you know
- Make some assumptions linking what you know to the thing you want to learn about
- Use the model output to learn about the thing you want to learn about
- Evaluate the model and its sensitivity to assumptions
A recipe for modeling
Example: how many people inject drugs?
Question: How many people inject drugs (e.g. opioids) in my city?
Data: counts of \(m\) individuals’ emergency room visits for overdose, \(X_1,\ldots,X_m\), all positive, for one unit of time (e.g. year). We only see \(X_i\) if person \(i\) had at least one overdose.
Why is this a hard problem?
Example: a simple model (part 1)
- State the thing you want to learn about
Let \(N\) be the number of people who inject drugs in the city
- State what you know
Let \(X_1,\ldots,X_N\) be the number of times each has overdosed and been taken to the emergency room.
Let \(M=m\) the number who have had at least one overdose, and we know \(X_1,\ldots,X_m > 0\)
Example: a simple model (part 2)
- Make some assumptions linking what you know to the thing you want to learn about
Assume every drug injector has an overdose with constant rate \(\lambda\) per unit time.
- Use the model output to learn about the thing you want to learn about
Therefore, \[ X_i \sim \text{Poisson}(\lambda) \] independently for each \(i=1,\ldots,N\). The distribution of \(X_i\) for \(X_i>0\) is \[ \Pr(X_i=k) = \frac{\lambda^k e^{-\lambda}}{k! (1-e^{-\lambda})} \] So we can estimate \(\lambda\) from the observable data.
Example: a simple model (part 3)
Then, we know that \[ E[M] = \Pr(X_i>0) \times (\text{number at risk}) \] and so \[ E[M] = (1-e^{-\lambda}) N . \]
Rearranging, we have
\[ \hat{N} = \frac{m}{1-e^{-\hat\lambda}} \]
- Evaluate the model and its sensitivity to assumptions (saved for later)
Example: post-mortem
Where was the magic step?
A common distributional assumption for positive and zero \(X_i\)’s,
\[ X_i \sim \text{Poisson}(\lambda) \]
This allowed estimation of \(\lambda\), and implies that
\[ M \sim \text{Binomial}(N,1-e^{-\lambda}) \]
which we can use to estimate \(N\).
Some questions:
- What are the weaknesses of this model?
- Which assumptions are likely to be violated in practice?
- Does it matter if we get the Poisson assumption exactly right?
- If not, are we likely to over- or under-estimate \(N\)?
The big picture
Why are mechanistic models useful?
- Intuitive: they formalize hypotheses
- Statistical: they limit free parameters
- Interpretability: parameters have real-world meaning
Why are mechanistic models sometimes dangerous?
- Limit hypotheses to models that are easy to specify
- Inflexible structure limits fitting
- Sometimes you don’t know when the mechanism is wrong, even when they fit data well
- More complicated reasons related to causal inference
Statistical vs mechanistic models
If you have taken a statistics class, you have seen statistical approaches to explaining variation. For example, consider the “statistical regression model” \[ y = \alpha + \beta x + \epsilon \] If we regard \(x\) as a treatment and \(y\) as a health outcome for a given patient, then we would like to think of \(\beta\) as the “effect” of the treatment.
This model posits a linear relationship between treatment and outcome. Given a one-unit change in \(x\), we expect the outcome \(y\) to change by an increment of \(\beta\).
My philosophy
I think there is no difference between “statistical” and “mechanistic” models, except for the stories we tell about their structure and coefficients. I think:
- We should strive to interpret statistical models in a mechanistic way, and reject them if they do not help us learn about the mechanism of interest.
- We should treat mechanistic models as statistical models and fit them to data, whenever possible. When not possible, we should ask what new data we ought to collect, or how to identify only the mechanistic features of interest.
This is not a statistics course, so we won’t say much more about balancing parsimony and realism for statistical inference.
References
Basu, Sanjay, and Jason Andrews. 2013. “Complexity in Mathematical Models of Public Health Policies: A Guide for Consumers of Models.” PLoS Medicine 10 (10). Public Library of Science: e1001540.
Stokey, Edith, and Richard Zeckhauser. 1978. Primer for Policy Analysis. WW Norton.